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Abstract 

Maximum likelihood estimation procedures for the Jelinski-Moranda 
software reliability model often give misleading answers. We show here 
that a reparameterization and a Bayesian analysis eliminate some of the 
problems incurred by MLE methods and often give better predictions on 
sets of real and simulated data. 

Practical difficulties in estimating the initial number of errors 
N and the failure rate of each error (}) by the method of maximum like- 
lihood are: 

/\ 

1. N , the MLE of N , is occasionally infinite (i.e., the routines 

/S 

for calculating N and (j) do not converge) . Littlewood and 
Verrall show that N is finite if and only if the regression 
line of the interevent times t^ vs. i has positive slope. 

2. A serious problem is that often N - n , the sample size, and 
sometimes N = n . Thus the MLE predicts that the program is 
perfect even when it is far from being so. Forman and Singpur- 
walla have shown that N and <f) can only be trusted near the 
end of debugging, i.e., when almost all failures have been 
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removed . 



3. Even when these problems are not encountered, the results 
obtained from the model are too optimistic; it predicts the 
reliability to be greater than it really is. 

In view of these deficiencies, we are led to consider a Bayesian 
approach to the estimation problem. It seems plausible that it is easier 
to correctly estimate the initial program failure rate X = N(() than the 

/N 

initial number of bugs N , since small errors in <p could lead to large 
errors in N . It is therefore plausible to reparameterize the model to 
(X,(j)) instead of (N,<})) . 

Using now the Bayesian approach, letting prior (X,(J>) = prior (X)* 
prior (c()) , where prior (X) and prior (cj)) are gamma distributed, and 
using 


Vi<'> ' '■(Vi ^ ^ ' 


'i V 


= / < t 1 x,4>) post(X,(f) I t^,. . . ,t^)dXd(j> 

we obtain an explicit estimate of the program's current reliability. 
Similarly, we can get in closed form the distributions of the number of 
bugs remaining in the program, the number of bugs that have to be removed 
in order to attain a given reliability, and the times between future 
consecutive failures (provided they are well defined, i.e., the program 
is not perfect). 

The quality of these estimations was examined for the special case 
when X and tj) have an (improper) uniform prior distribution over 
[0,0°) (i.e., a noninfo rmative prior distribution). The predictions were 

examined both for real and for simulated sets of data. In all cases where 


ML erroneously predicts the program to be perfect, the Bayesian method 


gives a positive probability that the program is not perfect. Moreover, 
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since the predicted reliability is given in closed form, problems of 
convergence of the computer program are not encountered. 

To examine the quality of prediction, we use a goodness of fit pro- 
cedure. Suppose that from the data tj^,...,t^ we predict the distribu- 
tion of T , the time to next failure. We then observe t ,, . 

n+1 n+1 

Define U = Pr (T . , < t , , ) . If the model is correct, then U are 
n n+1 n+1 n 

uniform variables on (0,5,) . We compare the sample c.d.f. of the u 's 

n 

with a line of unit slope which is the uniform c.d.f. 

When applying the goodness of fit procedure to real data sets, the 
Bayesian approach is almost always better than the MLE method. For the 
simulated data, the goodness of fit procedure on the Bayesian estimates 
give very good results; this, however, is not always true for the real 
data sets. 

There seems to be evidence that the J-M model is intrinsically opti- 
mistic in its estimate of software reliability. This could be a conse- 
quence of the assumption that all errors contribute equally to the failure 
rate. A new model by Littlewood relaxes this assumption with the result 
that earlier fixes tend to involve larger reductions in the failure rate 
than the later ones. It can be shown that this model is less optimistic 
than the J-M model and we hope to examine its performance on real and 
simulated data in future work. 
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THE VIEWGRAPH MATERIALS 
for the 

B. LITTLEWOOD/A. SOFER PRESENTATION FOLLOW 
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JELINSKI-MORANDA model assumptions: 

1 . Successive inter-failure times Tj , T 2 , . . . . are independent. 

pdf(tjlXi)-Xj e-Vi 

2. Xj = (N - i + 1 ) 0 where 

N is “initial number of faults” 0 is “contribution to program failure rate from each fault” 


f.r 



time 


Note that 

1. All fixes have same effect. 

2. Same model by SHOOMAN and MUSA. Same assumptions 
for NHPP model by GOEL-OKUMOTO. 
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There seems to be 3 problems with J-M: 

^ A 

1 . N occassionally infinite (0 = 0) 

Nec. & Suff. conditions: “Regression line of t versus i has negative slope” 
(Littlewood, Verrall: 1981IEEETR) 

This can also occur with simulated data from J-M with finite N, 0 9^ 0, 

A X 

However X = N0 is finite, non-zero. 

2. Reliability predictions always(?) too optimistic 

A 

3. N usually too small, sometimes equal to sample size (i.e. program is “perfect”) 
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Table 7. 

Failure Intervals — System 3 System Test Phase 


i 

Ti 


1 

115, 

1 

2 

0, 

1 

3 

83, 

3 

4 

178, 

3 

5 

194, 

3 

6 

136, 

3 

7 

1077, 

3 

8 

15, 

3 

9 

15, 

3 

10 

92, 

3 

11 

50, 

3 

12 

71, 

3 

13 

606, 

6 

14 

1189, 

8 

15 

40, 

8 

16 

788, 

18 

17 

222, 

18 

18 

72, 

18 

19 

615, 

18 

20 

589, 

26 

21 

15, 

26 

22 

390, 

26 

23 

1863, 

27 

24 

1337, 

30 

25 

4508, 

36 

26 

834, 

38 

27 ^ 

3400,, 

_40 

28 

6, 

40 

29 

4561, 

42 

30 

3186, 

44 

31 

10571, 

47 

32 

563, 

47 

33 

2770, 

47 

34 

652, 

48 

35 

5593, 

50 

36 

11696, 

54 

37 

6724, 

54 

38 

2546, 

55 

39 

-10175, 

56 
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SYSTEM 3 
FAILURE 

A 

N 

ESTIMATED 

A 

$ 

ESTIMATED INITIAL MORANDA 

NUMBER 

FAILURES 

MTTF 

PHI 

2 

999999 

0.5750E+02 

0. 173913E-07 

3 

999999 

0.6600E+02 

O. 151515E~07 

4 

5 

0.5900E+02 

0.338983E'02 

5 

6 

0.6480E+02 

0. 257202E-02 

6 

8 

0.7275E+02 

0. 171821E-02 

7 

7 

0.7884E+02 

0. 181206E~02 

8 

8 

0. 8845E+02 

0. 14 1318E-02 

9 

12 

0. 1 196E+03 

0. 696972E-03 

10 

19 

0. 1396E+03 

0. 377017E~03 

1 1 

55 

0. 1609E+03 

0. 1 12990E~03 

12 

999999 

O. 1688E+03 

0. 592304E-08 

13 

22 

0. 1387E+03 

0. 327621E-03 

14 

15 

0. 1 125E + 03 

0.592367E-03 

15 

18 

0. 1306E + 03 

0.425447E'03 

16 

18 

0. 1306E^03 

0. 425294E~03 

17 

21 

0. 1476E+03 

0.322715E'03 

18 

25 

O. 1616E+03 

0. 247463E-03 

19 

25 

0. 1622E+03 

0.246615E~03 

20 

25 

0. 1612E+03 

0.248210E'03 

21 

31 

0. 1807E+03 

0. 178535E'03 

22 

33 

0. 1854E + 03 

0, 1634 13E~03 

23 

26 

0. 1609E + 03 

0.239046E-03 

24 

26 

0. 1606E+03 

0.239457E-03 

25 

25 

0. 1520E+03 

0. 263205E-03 

26 

26 

O. 1628E+03 

0.236199E-03 

27 

27 

0. 1764E+03 

0.210001E-03 

28 

28 

0. 1876E+03 

0. 190384E-03 

29 

29 

O.2023E+03 

0. 170456E-03 

30 

30 

0.2182E+03 

O. 152766E-03 

31 

31 

0.2427E+03 

0. 132935E-03 

32 

32 

O.2642E+03 

0. 1 18265E-03 

33 

33 

O.2853E+03 

0. 106202E-03 

34 

34 

0.304 1E+03 

0.967196E'04 

35 

35 

O.3248E+03 

0.87955$E'04 

36 

36 

0.3519E+03 

0.789439E-04 

37 

37 

O.3804E+03 

0.710397E-04 

38 

38 

0.4073E+03 

0. 64604 1E'04 
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How well does model perform? 


Simplest problem is estimation of current reliability: 

Given data tj , . . tj_j , what can we say about Tj? 

What is cdf Fj (t)? 

Obtain ML estimates of N, 0, based on tj, . . > t^^_j and use “Predictor distribution” 

Fj(t)= 1 

If prediction is “good” 

Uj = Fj (Tj ) is approx. 

U(0,1). Examine Q-Q plots of realizations 
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EXAMPLE 


Data: MUSA “System 1”, range of i:30-129 
Jelinski-Moranda: poor prediction, optimistic 
Littlewood-Verrall: good prediction, slight pessimism 
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Bayesian J-M 


Reparameterize to (X, 0) from (N, 0) where X = N(^ “initial failure rate”. 

Assume: 

prior (X, 0) = prior (X) • prior (0) where prior (X) and prior (0) are gamma distributed 
Then “predictor distribution” is 

Fi (t) - P(Ti < t) = P(Tj < t I tj , . . . Vi) = / P(Ti < t 1 X,0) post (X,0 1 tj. . Vi ) dXd0 


Reparameterization: Informal Justification 



i, FAILURE NUMBER 
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For the case of uniform (improper) priors we get: 

Fi+1 (t'V ti) = 


‘i aj^jk!(i-k)! / I I 

XI I 


(S (i - j + 1) (t + it.y 

i-1 aj^ i - 1 (i-k)! 

where c * = Y] ~j‘ i 

^ (S (i-j)t/+l (2 tj)‘-^+l 

'f=l J Fi J 


i-k+1 


and where a^ j is the coefficient of x‘ in fl (x + k) = ; x 

k=» ^ 

These coefficients are easily computed from the relation 
^01 ^ 1 ^ * ^)i ^ ^ "^i 


J-k 


DATA: MUSA "System 3", i=18. .37 

" J-M ML estimation of (N,0) 

J-M Bayes, uniform (improper) priors on (X, (^) 
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Conclusion! 


1. Bayes J-M seems always (?) better than MLE J-M, but sometimes only slightly. 


2. Results on real data are always optimistic. 


3. But on SIMULATED data from J-M model, Bayes is very good, ML poor 


real data do not follow J-M model? 
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Hypothesis: Assumption of equal (t>'s is wrong. In fact <^'s different. 

Larger ones tend to be eliminated earlier: 


f.r. 
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